{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Region Proposal Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Anchor Box\n",
    "In `configuration.py` we have\n",
    "```python\n",
    "aspect = lambda s, r: (s * 1 / r ** 0.5, s * r ** 0.5)\n",
    "self.rpn_base_aspect_ratios = [\n",
    "           [(1,1), aspect(2**0.5,2), aspect(2**0.5,0.5),],\n",
    "           [(1,1), aspect(2**0.5,2), aspect(2**0.5,0.5),],\n",
    "           [(1,1), aspect(2**0.5,2), aspect(2**0.5,0.5),],\n",
    "           [(1,1), aspect(2**0.5,2), aspect(2**0.5,0.5),],\n",
    "        ]\n",
    "\n",
    "self.rpn_base_sizes = [8, 16, 32, 64]\n",
    "self.rpn_scales = [2, 4, 8, 16]\n",
    "```\n",
    "In which `s, r` means scale and ratio, each element in `rpn_base_apsect_ratios` means (height, width) of one anchor box. 4 array means 4 layers, so there are 3 different anchor boxes for each of the 4 feature maps generated from FPN, i.e\n",
    "$$ \n",
    "\\begin{align}\n",
    "height & = scale \\times (\\frac{1}{\\sqrt{ratio}}) \\\\\n",
    "width  & = scale \\times (\\sqrt{ratio})\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "So\n",
    "$$\\frac{width}{height} = ratio$$\n",
    "\n",
    "And we have 2 scales $(1, \\sqrt{2})$ and 3 aspects $(1:1, 1:2, 2:1)$, each feature map correspinding to 3 different anchor boxes.\n",
    "\n",
    "![](images/anchor.svg)\n",
    "\n",
    "1. The actual proposal size is controlled by `rpn_base_sizes`, different feature maps have different receptive field, small feature map has big receptive field, so we assign big base size of anchor box for small feature maps. \n",
    "\n",
    "2. The corresponding scale ratio from image to feature map is recorded in `rpn_scales`, when using RoiAlign, we need to know which feature map to pool from given proposals by measuring the proposals' sizes. When feature map is set, we need these scales to get the actual sizes to crop from the feature maps.\n",
    "\n",
    "The figure above explains the parameters used in `configuration.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RPN's output\n",
    "\n",
    "![](images/rpn_head.svg)\n",
    "\n",
    "1. FPN generate 4 feature maps from a image\n",
    "2. For one feature map say p3(64\\*64), take each pixel(of the feature map) as the centre of an anchor box. \n",
    "3. There are a total of $(64*64)*3*2 = 24576$ proposals arranged in shape $(3, 2, 64, 64)$, in which 2 means the score for foreground/background. Flatten it on the pixels' dimension gives the tensor $(B, N, 2)$, in which `B` is batch size, `N` will be calculated:\n",
    "4. All 4 feature maps have a total of $(16*16*3 + 32*32*3 + 64*64*3 + 128*128*3) = 65280$ anchor boxes for foreground and background, so the final return size for RPN layer is $(B, 65280, 2)$. Let's check it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "import torch\n",
    "from configuration import Configuration\n",
    "from net.layer.rpn.rpn_head import RpnMultiHead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "cfg = Configuration()\n",
    "# set anchor boxes to our example above\n",
    "aspect = lambda s, r: (s * 1 / r ** 0.5, s * r ** 0.5)\n",
    "cfg.rpn_base_aspect_ratios = [\n",
    "           [(1,1), aspect(2**0.5,2), aspect(2**0.5,0.5),],\n",
    "           [(1,1), aspect(2**0.5,2), aspect(2**0.5,0.5),],\n",
    "           [(1,1), aspect(2**0.5,2), aspect(2**0.5,0.5),],\n",
    "           [(1,1), aspect(2**0.5,2), aspect(2**0.5,0.5),],\n",
    "        ]\n",
    "\n",
    "# input features, the batch size is set to 5, channels of each feature map is 256\n",
    "p2 = torch.randn(5, 256, 128, 128)\n",
    "p3 = torch.randn(5, 256, 64, 64)\n",
    "p4 = torch.randn(5, 256, 32, 32)\n",
    "p5 = torch.randn(5, 256, 16, 16)\n",
    "fs = [p2, p3, p4, p5]\n",
    "# init RPN head\n",
    "net = RpnMultiHead(cfg, 256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([5, 65280, 2]) torch.Size([5, 65280, 2, 4])\n"
     ]
    }
   ],
   "source": [
    "logits_flat, deltas_flat = net(fs)\n",
    "print(logits_flat.size(), deltas_flat.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RPN layer has 2 branch, one gives the score for foreground/background for each anchor box, another gives the box regression parameters, which is used to reshape each anchor box to fit the instance better. There are 4 parameters to refine the anchor box, so we have output tensor of size $(B, 65280, 2, 4)$ \n",
    "\n",
    "So this is the base idea behind RPN layer, which is probably the most complicated layer in mask rcnn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RPN Train/Test Process\n",
    "\n",
    "![](images/rpn.svg)\n",
    "\n",
    "1. Get features from FPN\n",
    "2. Get rpn output: logits and deltas\n",
    "3. Make targets for training from `rpn_target.py`\n",
    "4. Do nms to get rpn proposals, which is in `nms.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RCNN Train/Test Process\n",
    "\n",
    "![](images/rcnn.svg)\n",
    "\n",
    "The train/test process for rcnn is not much different from RPN except for:\n",
    "\n",
    "1. RCNN use cropped features by roi-align layer. The crop size is decided by rpn proposals\n",
    "2. RCNN proposals for training are re-sampled, with balanced positive/negative samples\n",
    "3. RCNN predict class label for each proposal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MASK Train/Test Process\n",
    "\n",
    "![](images/mask.svg)\n",
    "\n",
    "The train/test process for mask head is not much different from RPN and RCNN except for:\n",
    "\n",
    "1. Mask head predicts per-pixel probability mask for each class, so it's output is (B, mask_size, mask_size)\n",
    "2. The target for mask size is cropped from truth box and truch instance, which may not the same size as mask_size, so we need to resize the cropped instance into mask_size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
